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Program Description^ Read Well® is a reading curricuium for kindergarten and first-grade 

students whose goal is to increase students’ iiteracy abiiities. The 
program provides instruction in phonemic awareness, phonics, 
vocabulary, comprehension, and fiuency. Students are given 
opportunities to discuss the vocabuiary concepts that are presented 
in each story. The program is based on the tenets of scaffolded 
instruction, in which teachers begin by presenting models and 


gradually decreasing their support by providing guided practice 
before students are asked to complete the skill or strategy indepen- 
dently. For example, the student and teacher read new text aloud 
with the teacher reading the difficult or irregular words. As student 
skills (and motivation) increase, the amount of teacher-read text 
decreases and the student is given greater independence. The pro- 
gram combines daily whole class activities with small group lessons. 



Research^ One study of Read Well® that falls within the scope of the 
Engiish Language Learners review protocol meets the What 
Works Clearinghouse (WWC) evidence standards. This study, 
which included 34 first-grade English language learner students 
from one school in rural Colorado, examined program impacts 
on students’ reading and Engiish ianguage deveiopment.'* 



Based on this one study, the WWC considers the extent of 
evidence for Read Well® on English language learners to be 
small for both reading achievement and English language devel- 
opment. No studies that meet WWC evidence standards with or 
without reservations examined the effectiveness of Read Well® 
on English language learners in mathematics achievement. 



1. This report has been updated to include reviews of four studies that were not included in the earlier review of Read Well®. Of these studies, two are not 
within the scope of the protocol, one is within the scope of the protocol but does not meet evidence standards, and one meets standards. One study 
that meets standards with reservations in the earlier review (Denton, Parker, & Hasbrouck, 2004) no longer meets evidence standards because the 
intervention and comparison groups are not shown to be equivalent at baseline. (The protocol for the English Language Learners topic area was revised 
to specify that groups must be equivalent on the pretest for a quasi-experimental design.) A complete list and disposition of all studies reviewed is 
provided in the references. 

2. The descriptive information for this program was obtained from a publicly available source: the program’s website (http://store.cambiumlearning.com/ 
resource. aspx?page=ProgramOverview&site=sw&parentld=01 9005451, downloaded May 2009). The WWC requests developers to review the program 
description sections for accuracy from their perspective. Further verification of the accuracy of the descriptive information for this program is beyond 
the scope of this review. The literature search reflects documents publicly available by February 2009. 

3. The studies in this report were reviewed using WWC Evidence Standards, Version 2.0 (see the WWC Procedures and Standards Flandbook, Chapter III), 
as described in protocol Version 2.0. 

4. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 
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Eff6CtiV6n6SS Read Well® was found to have no discernible effects on reading achievement and potentially positive effects on English language 
development for elementary school English language learners. 



Reading 


Engiish ianguage 


Mathematics 


achievement 


deveiopment 


achievement 



Rating of effectiveness 


No discernible effects 


Potentially positive effects 


na 


Improvement index® 


Average: -1 percentile points 


Average: +21 percentile points 


na 



Range: -2 to -1 percentile points na 

na = not applicable 



Additional program Developer and contact 

information Developed by Sopris West Educational Services, Read Well® is 
distributed by Cambium Learning, Inc. Address: Sopris West, 
4185 Salazar Way, Frederick, CO 80504. Email: customerservice® 
sopriswest.com. Web: http://store.cambiumlearning.com. 
Telephone: (303) 651-2829 or (800) 547-6747. 



skills (and motivation) increase, the amount of teacher-read text 
decreases and the student is given greater independence. 

The Read Well® instructor package includes all teacher materi- 
als and one set of student materials necessary for implementing 
the program in a classroom. A professional development program 
is available for teachers interested in using the curriculum. 



Scope of use 

Read Well® has been implemented in various regions of the 
country with kindergarten, first-grade, and second-grade regular 
education students and English language learners. 

Teaching 

The Read Well® program involves explicit, systematic instruction 
in English language decoding, sustained practice of skills in 
decodable text, and frequent opportunities to discuss vocabu- 
lary and concepts presented in text. Reading and writing skills 
include story maps, story retells, and guided reports. A key fea- 
ture of Read Well® is the use of scaffolded instruction, in which 
teachers begin by presenting models and gradually decreasing 
their support by providing guided practice before students 
are asked to complete the skill or strategy independently. For 
example, the student and teacher read new text aloud with 
the teacher reading the difficult or irregular words. As student 



Cost® 

Read Well® can be purchased in whole class, small group, or 
student packages that are targeted to kindergarten, first-grade, 
or second-grade students. The whole class instructor package for 
kindergarten students costs $1,253.95. It includes both whole class 
and small group instructional components. The cost for a stand- 
alone small group instructor package is $511.95. The student pack- 
age costs $1,880.95 for 24 students and $428.49 for six students. 

The instructor package for first-grade students costs 
$1,044.95. The student package costs $1,985.95 for 24 students 
and $574.95 for six students. 

The instructor package for second-grade students costs 
$1,097.49. The student package costs $1,985.49 for 24 students 
and $574.95 for six students. 

Materials in the instructor and student packages are also 
available for individual purchase at prices ranging from $19.95 
to $396.95. 



5. These numbers show the average and range of student-level Improvement Indices for all findings across the study. 

6. The cost of Read Well® was received from the developer. Cambium Learning, Inc. in May 2009. 
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Standards Handbook, Appendix G). The extent of evidence 
takes into account the number of studies and the total sample 
size across the studies that meet WWC evidence standards with 
or without reservations.® 

The WWC considers the extent of evidence for Read Well® 
to be small for both reading achievement and English language 
development for English language learners. No studies that meet 
WWC evidence standards with or without reservations examined 
the effectiveness of Read Well® in mathematics achievement for 
English language learners. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the WWC Procedures and 



RSSSarch Five studies reviewed by the WWC investigated the effects of 
Read Well® on English language learners. One study (Frasco, 
2008) is a randomized controlled trial that meets WWC evidence 
standards.^ The study included 34 English language learner stu- 
dents In first grade. The intervention group included 17 students 
assigned to receive Read Well® as their core reading program. 
The control group included 17 students assigned to receive 
MacmIllan/McGraw-HIII 2003 as their core reading program. 

The remaining four studies do not meet either WWC evidence 
standards or eligibility screens. 



Effectiveness Findings 

The WWC review of interventions for English language learn- 
ers addresses student outcomes in three domains: reading 
achievement, English language development, and mathematics 
achievement. The studies included in this report cover only 
two domains: reading achievement and English language 
development. The findings that follow present WWC-calculated 
estimates of the size and the statistical significance of the effects 
of Read Well® on English language learners.® 

Reading achievement. Frasco (2008) reported a not statisti- 
cally significant difference in reading gains, as measured by the 



Dynamic Indicators of Basic Literacy Skills (DIBELS) Nonsense 
Word Fluency subtest, and a not statistically significant differ- 
ence in fluency and comprehension gains, as measured by the 
Gray’s Oral Reading Test-Fourth Edition (GORT-4).^° The average 
effect size across these outcomes was not large enough to be 
considered substantively Important according to WWC criteria. 

English language development. Frasco (2008) reported a posi- 
tive and statistically significant difference in vocabulary gains 
as measured by the Peabody Picture Vocabulary Test-Third 
Edition (PPVT-lll). However, WWC calculates significance based 
on posttest standard deviations (as opposed to the study, which 



7. In addition to the results discussed below, Frasco (2008) also reports results on the Comprehensive Test of Phonological Processing (CTOPP) that are 
excluded from this report since the test measures phonemic awareness, which is not part of any English Language Learners domain. 

8. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined for Read Well* is in Appendix A5. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate 
the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, 
Appendix D for multiple comparisons. For the Read Well* studies summarized here, no corrections for clustering or multiple comparisons were needed. 
However, the WWC calculates statistical significance based on posttest standard deviations and not on pretest to posttest gains; this adjustment was 
applied to the Frasco (2008) study. 

10. The study also includes results on the Comprehensive Test of Phonological Processing. The English Language Learners area does not consider phonological 
awareness measures as part of the reading achievement or English language development domains. Therefore, these results are not included in this review. 
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EffSCtivSneSS (continued) used standard deviations of pretest to posttest gains): after this 

adjustment, the WWC found that the difference in vocabulary 
gains was not statisticaiiy significant. The effect size is iarge 
enough to be considered substantiveiy important according 
to WWC criteria. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentialiy positive, mixed, no discernible 



effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statisticai significance of the findings, the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Procedures and Standards Handbook, 
Appendix E). 



The WWC found Read Well® 
to have no discernible 
effects for reading 
achievement and potentially 
positive effects for English 
language development for 
English language learners 



Improvement index 

The WWC computes an improvement index for each individuai 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see WWC Proce- 
dures and Standards Handbook, Appendix F). The improvement 
index represents the difference between the percentiie rank 
of the average student in the intervention condition and the 
percentile rank of the average student in the comparison condi- 
tion. Uniike the rating of effectiveness, the improvement index is 
entireiy based on the size of the effect, regardless of the statisti- 
cal significance of the effect, the study design, or the analysis. 
The improvement index can take on values between -50 and 
+50, with positive numbers denoting favorable results for the 
intervention group. 



The average improvement index for reading achievement 
is -1.46 percentile points across the study, with a range of 
-2.30 to -0.61 percentile points across findings. The average 
improvement index for English language development is +21.06 
percentile points across the study. 

Summary 

The WWC reviewed five studies on Read Well® for English language 
learners. One of these five studies meets WWC evidence standards: 
the remaining four studies do not meet either WWC evidence 
standards or eligibility screens. Based on the one study, the WWC 
found no discernible effects in the reading achievement domain 
and potentially positive effects in the English language development 
domain for English language learners. The conclusions presented 
in this report may change as new research emerges. 



Refsrsncss Meets WWC evidence standards 

Frasco, R. D. (2008). Effectiveness of Reading First for English 
language learners: Comparison of two programs (Doctoral 
dissertation, Walden University, 2008). Dissertation Abstracts 
International, 69(03A), 141-879. 

Studies that fall outside the English Language Learners 
review protocol or do not meet WWC evidence standards 

Denton, C. A., Anthony, J. L., Parker, R., & Hasbrouck, J. E. 
(2004). Effects of two tutoring programs on the English 



reading development of Spanish-English bilingual students. 
The Elementary School Journal, 104(A), 289-305. The study 
does not meet WWC evidence standards because it uses a 
quasi-experimental design in which the analytic intervention 
and comparison groups are not shown to be equivalent. 
Additional source: 

Denton, C. A. (2000). The efficacy of two English interventions 
in a bilingual education program (Doctoral dissertation, 
Texas A&M University, 2000). Dissertation Abstracts Inter- 
national, 67(11), 4325A. (UMI No. 9994233) 
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RsfsrsnCGS (continued) Kamps, D., Abbott, M., Greenwood, C., Arreaga-Mayer, C., 

Wills, H., Longstaff, J., et al. (2007). Use of evidence-based, 
small-group reading instruction for English language learners 
in elementary grades: Secondary-tier intervention. Learning 
Disability Quarterly, 30(3), 153-168. This study does not meet 
WWC evidence standards because the measures of effective- 
ness cannot be attributed solely to the intervention— the 
intervention was combined with another intervention. 

Santoro, L. E., Jitendra, A. K., Starosta, K., & Sacks, G. 

(2006). Reading well with Read Well: Enhancing the reading 



performance of English language learners. Remedial & 
Special Education, 27(2), 105-115. The study is ineligible for 
review because it does not use a comparison group. 

Sopris West Educational Services. (2007). Read Well: Results 
with the Read Well curriculum, kindergarten and first grade, 
Huntsville, Alabama. Frederick, CO: Author. The study Is Ineli- 
gible for review because it does not examine an intervention 
implemented In a way that falls within the scope of the review. 
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Appendix 



Appendix A1 Study characteristics: Frasco, 2008 



Characteristic 


Description 


Study citation 


Frasco, R. D. (2008). Effectiveness of Reading F/rsf for English language learners: Comparison of two programs (Doctorial dissertation, Walden University, 2008). Dissertation 
Abstracts Intemationai, 69(03A), 141-879. 


Participants 


The study was based on 36 first-grade English language learner students. Seventeen of these students were randomly assigned to the Read Weil® intervention group and 
19 were assigned to the Macmillan/McGraw-Hill control group. Two students attrited from the control group, resulting in 17 intervention group students and 17 control group 
students for a total of 34 students in the analysis sample. Overall attrition was 5.6% and differential attrition was 10.5%. 


Setting 


The study took place in a rural elementary school in eastern Colorado. English language learners constituted 61% of the school population. Eighty-two percent of the study 
body qualified for free and reduced-price lunch during the 2007-08 academic school year. 


Intervention 


Eor approximately three months, students received a minimum of 90 minutes of daily instruction in Read Well® 1, which was utilized as their core reading program. The pacing 
depended on the level of mastery for each individual participant, allowing students to accelerate or slow down according to their grasp of the material. 


Comparison 


Students in the control group were taught reading using Macmillan/McGraw-Hill 2003 core reading program and also received 90 minutes of daily instruction. The pacing of 
the program is based on completing a story weekly, with intervention provided for students after giving the unit test during Week 6 of each unit. This study encompassed two 
skills-based units, or 12 weeks of instruction. Teachers used lesson maps and templates that included recommendations for where to replace or add activities for struggling 
readers such as English language learner students. 


Primary outcomes 
and measurement 


Study measures in the reading achievement domain included the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Nonsense Word Eluency Subtest and Gray’s Oral 
Reading Test-Eourth Edition (GORT-4). Study measures in the English language development domain included the Peabody Picture Vocabulary Test-Third Edition (PPVT-III). 
The study also reports results on the Comprehensive Test of Phonological Processing (CTOPP) that are excluded from this report, since the test measures phonemic aware- 
ness, which is not part of any English Language Learners domain. Eor a more detailed description of these outcome measures, see Appendices A2.1-A2.2. 


Staff/teacher training 


Three teachers and four instructional aides were involved in the study. Teachers and instructional aides received professional development to implement the program. 
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Appendix A2.1 Outcome measures for the reading achievement domain 



Outcome measure 


Description 


Dynamic Indicators of 
Basic Early Literacy 
Skills (DIBELS) Nonsense 
Word Fluency Subtest 


The DIBELS assessment is a screening tool used by teaohers to test students one-on-one to determine eariy iiteraoy skills. The Nonsense Word Fluency subtest is a one- 
minute probe that assesses the alphabetic principle. Students are given a list with consonant-vowel-consonant (CVC) and vowel-consonant (VC) nonsense words. Students 
are asked to orally produce each individual sound or read the whole nonsense word (as cited in Frasco, 2008). 


Gray’s Oral Reading 
Test-Fourth Edition (GORT-4) 


GORT-4 consists of norm-referenced tesfs of oral reading rafe, accuracy, fluency, and comprehension. In general, the entry point for grades 1 and 2 is Story 1. Each story has 
five oomprehension questions to answer following the oral reading of the given passage (as sited in Frasco, 2008). 


Appendix A2.2 Outcome measures for the Engiish ianguage deveiopment domain 


Outcome measure 


Description 


Peabody Picture Vocabulary 
Test-Third Edition (PPVT-III) 


The PPVT-III assesses receptive vocabulary. Students are shown pictures by the examiner. The examiner gives the student a vocabulary term, and the students identify the 
term by pointing to the picture or providing an oral response (as cited in Frasco, 2008). 
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Appendix A3.1 Summary of study findings inciuded in the rating for the reading achievement domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(students) 


Read Well® Comparison 

groups group 


Mean 

difference’* 
{Read Well® 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 



DIBELS: Nonsense Word 
Fluency Subtest 


Grade 1 


34 


72.00 

(17,34) 


Frasco, 2008® 

73,35 

(27.27) 


-1.35 


-0.06 


ns 


-2.30 


Gray’s Oral Reading Test- 
Fourth Edition 


Grade 1 


34 


12,00 

(3.39) 


12.06 

(4.18) 


-0,06 


-0,02 


ns 


-0.61 


Oomain average for reading achievement® 










-0.04 


na 


-1.46 



ns = not statistically significant 
na = not applicable 

DIBELS = Dynamic Indicators of Basic Early Literacy Skills 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the reading achievement domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. The standard deviations shown here differ from those shown in Frasco (2008), because the WWC uses posttest standard deviations, while the study reported stan- 
dard deviations of pretest to posttest gains. 

3. The mean of the intervention group reported here was calculated as the comparison group posttest mean plus the difference in pretest to posttest gains reported by the author. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The mean difference reflects the difference between groups 
in pretest to posttest gain scores calculated by the author. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. Effect sizes calculated based on data from Frasco (2008) use a difference-in- 
differences approach; that is, the numerator of the effect size is equal to the difference between the pretest-posttest mean difference for the intervention group and the pretest-posttest mean 
difference for the comparison group. The denominator reflects the pooled posttest standard deviation for the intervention and comparison groups. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The statistical significance reported here is not 
consistent with the author’s reported significance, since the WWC uses posttest standard deviations to provide the same metric across studies (as opposed to the study, which used standard 
deviations of pretest to posttest gains). 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Frasco (2008), no corrections for clustering or multiple comparisons were needed. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the Engiish ianguage deveiopment domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(students) 


Read Well® Comparison 

groups group 


Mean 

difference’* 
{Read Well® 
-comparison) 


Statistical 

Effect significance^ 

size® (at a = 0.05) 


Improvement 

index^ 











Frasco, 2008® 










PPVT-III 


Grade 1 


34 


89.12 

(17.65) 


81.41 

(7.52) 


7.71 


0.56 


ns 


21.06 



ns = not statistically significant 

PPVT-III = Peabody Picture Vocabulary Test-Third Edition 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the English language development domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. The standard deviations shown here differ from those shown in Frasco (2008) because the WWC uses posttest standard deviations, while the study reported stan- 
dard deviations of pretest to posttest gains. 

3. The mean of the intervention group reported here was calculated as the comparison group posttest mean plus the difference in pretest to posttest gains reported by the author. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The mean difference reflects the difference between groups 
in pretest to posttest gain scores calculated by the author. 

5. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. Effect sizes calculated based on data from Frasco (2008) use a difference-in- 
differences approach; that is, the numerator of the effect size is equal to the difference between the pretest-posttest mean difference for the intervention group and the pretest-posttest mean 
difference for the comparison group. The denominator reflects the pooled posttest standard deviation for the intervention and comparison groups. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. The statistical significance reported here is not 
consistent with the author’s reported significance, since the WWC uses posttest standard deviations to provide the same metric across studies (as opposed to the study, which used standard 
deviations of pretest to posttest gains). 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. 
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Appendix A4.1 Read Well® rating for the reading achievement domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of reading achievement, the WWC rated Read Well® as having no discernibie effects for Engiish ianguage iearners. 



Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: No studies showing a statisticaily significant or substantively important effect, either positive or negative. 

Met. The WWC anaiysis found no positive or negative statistically significant or substantively important effect. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Read Well® has only one study that met WWC evidence standards and no studies showing statistically significant positive effects. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. Read Well® has no studies showing statistically significant or substantively important negative effects. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. Read Well® does not have a study showing a statistically significant or substantively important positive effect. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing Indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. Read Well® does not have any studies showing a statistically significant or substantively important negative effect, but there is only one 
study showing indeterminate effects and none showing statistically significant or substantively important positive effects. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important pos/f/Ve effect. 

Not met. Read Well® has only one study showing indeterminate effects. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an Indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. Read Well® has only one study showing indeterminate effects. 



(continued) 
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Appendix A4.1 Read Well® rating for the reading achievement domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1; One study showing a statisticaily significant or substantively important negative effect and no studies showing a statisticaiiy significant or substantively 
important positive effect. 

Not met. No studies showing statistically significant or substantively important negative effects in this domain. 

OR 

• Criterion 2: Two or more studies showing statistically significant or substantively important negative effects, at least one study showing a statistically significant 
or substantively important pos/'f/Ve effect, and more studies showing statistically significant or substantively important negative effects than showing statistically 
significant or substantively important positive effects. 

Not met. No studies showing statistically significant or substantively important negative effects in this domain. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies showing statistically significant or substantively important negative effects in this domain. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. No studies showing statistically significant or substantively important positive effects in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings 
of potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Flandbook, Appendix E. 
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Appendix A4.2 Read Well® rating for the Engiish ianguage deveiopment domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the outcome domain of Engiish ianguage deveiopment, the WWC rated Read Well® as having potentially positive effects for Engiish ianguage learners. It did 
not meet the criteria for positive effects as it had oniy one study with a statisticaiiy significant and substantiveiy important positive effect. The remaining ratings (mixed 
effects, no discernibie effects, potentially negative effects, and negative effects) were not considered, as Read Well® was assigned the highest appiicabie rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At ieast one study showing a statisticaiiy significant or substantively important positive effect. 

Met. One study reviewed by the WWC reported a substantively important positive effect in Engiish ianguage development. 

AND 

• Criterion 2: No studies showing a statistically significant or substantiveiy important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statisticaiiy significant or substantively important positive effects. 

Met. The WWC analysis found no indeterminate, statistically significant negative, or substantively important negative effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Read Well® has only one study that met WWC evidence standards and showed a substantively important positive effect. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically or substantively important negative effects in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings 
of potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Flandbook, Appendix E. 
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Appendix A5 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schools 


Sample size 


Students 


Extent of evidence^ 


Reading achievement 


1 


1 




34 


Small 


English language development 


1 


1 




34 


Small 


Mathematics achievement 


na 


na 




na 


na 



na = not applicable/not studied 

1 . A rating of “medium to iarge” requires at ieast two studies and two schoois across studies in one domain and a totai sampie size across studies of at ieast 350 students or 14 ciassrooms. 
Otherwise, the rating is “smaii.” For more detaiis on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 
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